Does Registration Reduce Publication Bias? Evidence from Medical Sciences

There is broad recognition that reporting and publication biases operate at scale in academic scholarship. For example, published results in our field, political science, are much more likely to report p values just below the 0.05 critical value than just above it (Gerber and Malhotra, 2008a). This could be due to publication practices or to researcher degrees of freedom that allow them to select models that produce significant results. There are discussions inside the discipline, and more broadly in social sciences, about whether the introduction of a registration system like that prevailing in medicine would mitigate these effects, especially effects associated with researcher degrees of freedom. The basic idea is that if researchers precommit to a particular specification then they have less scope to (deliberately or inadvertently) report disproportionately results from analyses that yield significant findings (see for example Casey, Glennerster, and Miguel (2011); Humphreys, de la Sierra, and van der Windt (2013)). Though the arguments for registration are simple and strong, there is surprisingly little evidence that it makes any difference. Do registration requirements really change reporting norms and statistical practices? To contribute evidence to the debate, we seek to assess levels of bias before and after critical registration dates for journals that required and did not require registration. As such, this study aims to provide a systematic assessment of the impact of registration requirements on ‘critical value’ publication bias. The study faces two challenges however. We seek to understand the effects of registration on prospective experimental research yet our own analysis uses observational historical data. The fact that the data is observational makes it difficult to establish causal effects. We focus here on assessing whether patterns of published statistics are less indicative of bias after those journals adopted norms than before. Since the data is historical it is hard to pre-register our analysis in a credible way and to ensure that results we report are truly tests or are themselves fished results. These features generate interpretational difficulties for any results we find. We hope to gain clarity regarding what is learned from our analysis by surveying prior beliefs from readers about publication practices and what we should expect to see if registration did or did not have a causal effect.